## [1] "Excluded 11 participants based on catch-trial performance."
We further exclude participants who seem to provide random ratings independent of the scene that they are seeing. We quantify this by computing the mean rating for each utterance across all trials for each participant and computing the correlation between a participant’s actual ratings and their mean rating. A high correlation is unexpected and indicates that a participant chose ratings at random. We therefore also exclude the data from participants for whom this correlation is larger than 0.75.
## Warning in cor(rating_m, rating_m_overall): the standard deviation is zero
## [1] "Excluded 1 participants based on random responses."
We use the AUC function with the splines method to directly compute the AUC.
t-test and regression model with control variables:
##
## Two Sample t-test
##
## data: aucs.cautious$auc_diff and aucs.confident$auc_diff
## t = -0.31651, df = 180, p-value = 0.752
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -7.263266 5.255243
## sample estimates:
## mean of x mean of y
## 10.83339 11.83740
##
## Call:
## lm(formula = auc_diff ~ cond + test_order + first_speaker_type *
## test_order + confident_speaker, data = rbind(aucs.cautious,
## aucs.confident))
##
## Residuals:
## Min 1Q Median 3Q Max
## -44.853 -10.274 0.286 10.833 47.037
##
## Coefficients:
## Estimate Std. Error
## (Intercept) 22.171 3.007
## condconfident (might-biased) 1.004 2.497
## test_orderreverse 3.632 3.440
## first_speaker_typeconfidentfirst -27.217 3.564
## confident_speakerconfidentm -1.328 2.508
## test_orderreverse:first_speaker_typeconfidentfirst 1.483 5.006
## t value Pr(>|t|)
## (Intercept) 7.373 6.29e-12 ***
## condconfident (might-biased) 0.402 0.688
## test_orderreverse 1.056 0.293
## first_speaker_typeconfidentfirst -7.637 1.36e-12 ***
## confident_speakerconfidentm -0.529 0.597
## test_orderreverse:first_speaker_typeconfidentfirst 0.296 0.767
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 16.85 on 176 degrees of freedom
## Multiple R-squared: 0.3943, Adjusted R-squared: 0.3771
## F-statistic: 22.92 on 5 and 176 DF, p-value: < 2.2e-16